📬 Issue #: #1065

✍️ Description of changes:

Support Lambda Managed Instances / concurrent runtime (refs #1065)

This draft PR implements a concurrent runtime mode for Lambda Managed Instances, plus the API changes required in lambda-runtime and lambda-http. It preserves existing single-invocation behavior for classic Lambda. The implementation is based on the AWS guidance for building runtimes that support Lambda Managed Instances, as described in Building custom runtimes for Lambda Managed Instances.

Architecture (high level)

The runtime chooses between a classic "one invocation at a time" loop and a concurrent mode controlled by AWS_LAMBDA_MAX_CONCURRENCY:

• Sequential mode (default): a single /next long-poll loop fetches one event at a time and invokes the handler synchronously, preserving existing behavior.
• Concurrent mode (AWS_LAMBDA_MAX_CONCURRENCY >= 2): the runtime spawns AWS_LAMBDA_MAX_CONCURRENCY worker tasks. Each worker runs its own /next long-poll loop and processes invocations sequentially within that task, so overall concurrency is bounded by the number of workers. If a worker crashes, the runtime continues operating with the remaining workers and only terminates if it can't keep at least 1 worker alive.

sequenceDiagram
    participant Lambda as Lambda orchestrator
    participant Runtime as lambda-runtime
    participant Worker as Runtime worker task
    participant API as Runtime API
    participant Handler as User handler

    Lambda->>Runtime: Start process (env AWS_LAMBDA_MAX_CONCURRENCY = N)
    Runtime->>Worker: Spawn N worker tasks
    loop each worker (identical)
        loop each invocation
            Worker->>API: GET /runtime/invocation/next
            API-->>Worker: Invocation event + context headers
            Worker->>Handler: Call handler(event, context)
            Handler-->>Worker: Result or error
            Worker->>API: POST /response or /error
        end
    end

Breaking changes & compatibility

• Existing entrypoints (lambda_runtime::run, lambda_http::run, and lambda_http::run_with_streaming_response) keep their original signatures and sequential behavior. Handlers that compiled against the current release should continue to compile unchanged.
• New concurrent entrypoints are added:
  • lambda_runtime::run_concurrent
  • lambda_http::run_concurrent
  • lambda_http::run_with_streaming_response_concurrent
    These require handler services to implement Clone + Send + 'static (with responses/stream bodies Send/Sync + 'static) so they can be safely cloned and driven by the concurrent runtime.
• AWS_LAMBDA_MAX_CONCURRENCY is read directly by the runtime to decide concurrency and size the HTTP pool, without adding new public fields to Config (avoids semver breakage for struct literals).
• In concurrent mode (when the new entrypoints are used and AWS_LAMBDA_MAX_CONCURRENCY > 1), the runtime no longer sets _X_AMZN_TRACE_ID in the process environment. The per-invocation X-Ray trace ID is available via Context::xray_trace_id and tracing spans instead. Sequential mode behavior is unchanged.
• For the concurrent entrypoints, the behavior when AWS_LAMBDA_MAX_CONCURRENCY is set changes from "always sequential per environment" to "per-environment concurrency up to that value". Code that continues to call the existing run functions will remain strictly sequential even if the env var is set.

In other words: the earlier versions of this branch tightened the bounds on the existing run functions, but after maintainer feedback those entrypoints are left as-is and concurrency is opt-in via the new *_concurrent APIs.

Below is a concise summary of the changes (unfortunately many) by area.

Runtime & Config (lambda-runtime)

• Read AWS_LAMBDA_MAX_CONCURRENCY directly inside the runtime to decide whether concurrent mode should be enabled and to size the HTTP client pool, without extending Config.
• Runtime::new now sizes the lambda_runtime_api_client HTTP pool from max_concurrency so the number of idle connections matches expected concurrency.
• Runtime::run remains the original sequential /next loop via run_with_incoming, preserving existing behavior.
• New Runtime::run_concurrent spawns AWS_LAMBDA_MAX_CONCURRENCY worker tasks, each running its own /next loop. Sequential and concurrent modes share the same invocation processing helper, so there are not two separate event loop implementations to maintain. When Config::is_concurrent() is false, it falls back to the same sequential loop as Runtime::run.
• Worker failures are isolated: if a worker exits/panics, the runtime logs the failure and continues with fewer workers. The runtime terminates only if it can't keep at least 1 worker alive (i.e., all workers have exited).
• Unexpected clean worker exits are treated as infrastructure errors so the runtime returns an error once all workers are gone.
• Handler errors (user code returning Err) are reported to Lambda via /invocation/{id}/error and do not terminate the runtime.
• The existing graceful shutdown handler stays aligned with upstream. We do not attempt to preempt/coordinate in-flight invocations on SIGTERM (the expectation is that Lambda only shuts down environments when they are idle; local emulators may behave differently).
• Internal layers (api_client, api_response, trace) now implement/derive Clone so they can be composed into a cloneable service stack for the concurrent entrypoints.
• Context::new is more robust when lambda-runtime-client-context / lambda-runtime-cognito-identity headers are present but empty (treated as None instead of failing JSON parse).
• Added a regression test (concurrent_worker_crash_does_not_stop_other_workers) to verify worker isolation behavior.

HTTP & streaming (lambda-http, lambda-runtime-api-client)

• lambda_runtime_api_client::Client:
  • Added with_pool_size(usize) on the builder and threads a pool_size: Option<usize> into the Hyper client to set pool_max_idle_per_host.
  • Still works as before when pool_size is not provided.
• lambda_http::run and run_with_streaming_response keep their existing signatures and sequential behavior, delegating to lambda_runtime::run.
• New lambda_http::run_concurrent and lambda_http::run_with_streaming_response_concurrent wrap the same handler types but require them to be Clone + Send + 'static (with response/stream bounds aligned to lambda_runtime::run_concurrent) so they can be driven by the concurrent runtime.
• HTTP adapters (Adapter, StreamAdapter) are now Clone when the inner service is Clone, and the streaming path uses BoxCloneService internally for the concurrent entrypoint so the composed service stack can be cloned.
  • Note: run_with_streaming_response_concurrent is exported from lambda_http, similarly to run_with_streaming_response.

Tooling & examples

• Makefile and scripts/test-rie.sh:
  • Add RIE_MAX_CONCURRENCY and a test-rie-lmi target that runs RIE with AWS_LAMBDA_MAX_CONCURRENCY set, making it easy to exercise managed-instances behavior locally.
• examples/basic-lambda-concurrent:
  • New minimal example that calls lambda_runtime::run_concurrent so the concurrent code path can be exercised under RIE/LMI.
• examples/basic-lambda/src/main.rs:
  • Wraps lambda_runtime::run(func).await in an if let Err(err) block to log and propagate runtime errors when testing under RIE.

Validation

On this branch, I ran:

• cargo +nightly fmt --all -- --check
• cargo clippy --workspace --all-features -- -D warnings
• cargo +stable test --all-features -p lambda_runtime_api_client -p lambda_runtime -p lambda_http
• make test-rie EXAMPLE=basic-lambda
• make test-rie-lmi EXAMPLE=basic-lambda-concurrent (sets AWS_LAMBDA_MAX_CONCURRENCY inside the RIE container)

If maintainers prefer, this could be split into smaller PRs (e.g., builder/Config prep, handler Clone changes, and finally the concurrent runtime), but this branch shows the full "end-to-end" implementation so that it can be tested with Lambda Managed Instances.

🔏 By submitting this pull request

• I confirm that I've ran cargo +nightly fmt --all -- --check.
• I confirm that I've ran cargo clippy --workspace --all-features -- -D warnings.
• I confirm that I've made a best effort attempt to update all relevant documentation.
• I confirm that my contribution is made under the terms of the Apache 2.0 license.